Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.Comment: Shared first authors: Annika Reinke, Minu D. Tizabi; shared senior authors: Paul F. J\"ager, Lena Maier-Hei

Common Limitations of Image Processing Metrics:A Picture Story

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.Comment: This is a dynamic paper on limitations of commonly used metrics. The current version discusses metrics for image-level classification, semantic segmentation, object detection and instance segmentation. For missing use cases, comments or questions, please contact [email protected] or [email protected]. Substantial contributions to this document will be acknowledged with a co-authorshi

Visual Localization of Wireless Capsule Endoscopes Aided by Artificial Neural Networks

Various modalities are used for the examination of the gastrointestinal (GI) tract. One such modality is Wireless Capsule Endoscopy (WCE), a non-invasive technique which consists of a swallowable color camera that enables the detection of GI pathology with only minimal patient discomfort. Currently, tracking of the capsule position is estimated in the 3D abdominal space, using radio-frequency (RF) triangulation. The RF triangulation technique, however, does not provide sufficient information about the location of the capsule along the GI lumen, and consequently, the localization of any possible abnormality. Recently, we proposed a geometric visual odometry (VO) method for the localization of the capsule in the GI lumen. In this paper, we extend this state-of-art method by exploiting an artificial neural network (ANN) to augment the geometric method and achieve higher localization accuracy. The results of this novel approach are validated with an in-vitro experiment that provides ground truth information about the location of the capsule. The mean absolute error obtained, for a distance of 19.6cm, is 0.790.51cm. © 2017 IEEE

Visual Localization of Wireless Capsule Endoscopes Aided by Artificial Neural Networks

Various modalities are used for the examination of the gastrointestinal (GI) tract. One such modality is Wireless Capsule Endoscopy (WCE), a non-invasive technique which consists of a swallowable color camera that enables the detection of GI pathology with only minimal patient discomfort. Currently, tracking of the capsule position is estimated in the 3D abdominal space, using radio-frequency (RF) triangulation. The RF triangulation technique, however, does not provide sufficient information about the location of the capsule along the GI lumen, and consequently, the localization of any possible abnormality. Recently, we proposed a geometric visual odometry (VO) method for the localization of the capsule in the GI lumen. In this paper, we extend this state-of-art method by exploiting an artificial neural network (ANN) to augment the geometric method and achieve higher localization accuracy. The results of this novel approach are validated with an in-vitro experiment that provides ground truth information about the location of the capsule. The mean absolute error obtained, for a distance of 19.6cm, is 0.790.51cm

An artificial neural network architecture for non-parametric visual odometry in wireless capsule endoscopy

Wireless capsule endoscopy is a non-invasive screening procedure of the gastrointestinal (GI) tract performed with an ingestible capsule endoscope (CE) of the size of a large vitamin pill. Such endoscopes are equipped with a usually low-frame-rate color camera which enables the visualization of the GI lumen and the detection of pathologies. The localization of the commercially available CEs is performed in the 3D abdominal space using radio-frequency (RF) triangulation from external sensor arrays, in combination with transit time estimation. State-of-the-art approaches, such as magnetic localization, which have been experimentally proved more accurate than the RF approach, are still at an early stage. Recently, we have demonstrated that CE localization is feasible using solely visual cues and geometric models. However, such approaches depend on camera parameters, many of which are unknown. In this paper the authors propose a novel non-parametric visual odometry (VO) approach to CE localization based on a feed-forward neural network architecture. The effectiveness of this approach in comparison to state-of-the-art geometric VO approaches is validated using a robotic-assisted in vitro experimental setup. © 2017 IOP Publishing Ltd

Endoscopic single-image size measurements

In the practice of clinical gastrointestinal endoscopy, precise estimation of the size of a lesion/finding, such as a polyp, is quintessential in diagnosis, e.g. risk estimation for malignancy. However, various studies confirmed that endoscopic assessment of lesion size has inherent limitations and significant measurement errors. Image-based methods proposed for in-vivo-size measurements, rely on reference objects such as the endoscopic biopsy forceps. The aforementioned problem becomes more challenging in the field of capsule endoscopy, as capsules lack navigation and/or biopsy capabilities. To cope with this problem, we propose a methodology that requires only an endoscopic image - without any need for a reference object - in order to estimate the size of an object of interest in it. The first step in this methodology requires the user to define a linear segment within the image. Then, it takes into consideration the intrinsic parameters of the camera, to project known 3D points on the 2D image plane. With known 3D to 2D point correspondences, in order to perform a measurement, a rough approximation of the distance between the object of interest and the camera is needed. For this purpose, a convolutional neural network is utilized which generates depth maps from monocular images. The proposed methodology is validated by experimentation performed in a 3D printed model of the human colon. The results show that it is feasible to measure the size of various objects in endoscopic images with a mean absolute error of 1.10 mm ± 0.89 mm. © 2020 IOP Publishing Ltd

Deep Endoscopic Visual Measurements

Robotic endoscopic systems offer a minimally invasive approach to the examination of internal body structures, and their application is rapidly extending to cover the increasing needs for accurate therapeutic interventions. In this context, it is essential for such systems to be able to perform measurements, such as measuring the distance traveled by a wireless capsule endoscope, so as to determine the location of a lesion in the gastrointestinal tract, or to measure the size of lesions for diagnostic purposes. In this paper, we investigate the feasibility of performing contactless measurements using a computer vision approach based on neural networks. The proposed system integrates a deep convolutional image registration approach and a multilayer feed-forward neural network into a novel architecture. The main advantage of this system, with respect to the state-of-the-art ones, is that it is more generic in the sense that it is 1) unconstrained by specific models, 2) more robust to nonrigid deformations, and 3) adaptable to most of the endoscopic systems and environment, while enabling measurements of enhanced accuracy. The performance of this system is evaluated under ex vivo conditions using a phantom experimental model and a robotically assisted test bench. The results obtained promise a wider applicability and impact in endoscopy in the era of big data. © 2013 IEEE